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ABSTRACT 

The University of Puerto Rico Medical School 
conducted six-month Physician Retraining Programs (Curos de 
PerfecionamientoJ for two groups of foreign trained physicians 
starting in the Summer of 1970 and the Spring of 1971^ respectively. 
The characteristics of the 84 participants in these programs are 
examined in terms of pre-Curso medical knowledge and licensing 
indices and academic potential; Curso achievement indices; and 
post-Curso medical knowledge and licensing indices. Findings indicate 
that the physicians in the two programs: (1) were essentiality 
equivalent prior to Curso participation; (2) showed significant but 
differential gain during the programs; and (3) performed somewhat 
better on subsequent Puerto Rico licensing examinations tnan a group 
of foreign-trained physicians from the general population. All 
indications^ therefore^ suggest the success of the Curso in improving 
the medical knowledge of participants and improving the likelihood of 
their subsequent licensing (and thus better utilization in a sparse 
health manpower pool) . (Author) 
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Impact of the Curso cie Ferfeccionanucnto : 
An Audit of the I:f fectiveness of the 
Physician Retraining Program at the 
University of Puerto Rico 

ABSTRACT 

The University of Puerto Rico Medical School conducted six-month 
Physician Retraining Programs (Curso de PerfeccionamientoJ for two 
groups of foreign-trained physicians starting in the Summer of 1970 
and the Spring of 1971 respectively. The characteristics of the 
participants in these programs are examined in terms of pre~Curso 
medical knowledge and licensing indices and academic potenti^il; 
Curso achievement indices; and post-Curso medical knowledge and licensing 
indices , 

Findings indicate that the physicians in the two programs: (1) were 
essentially equivalent prior to Curso participation; (2) showed signifi- 
cant but differential gain during the programs; and (3) performed some- 
what better on subsequent Puerto Rico licensing examinations than a group 
of foreign— 'trained physicians from the general population. All indica- 
tions, therefore, suggest the success of the Curso in improving the 
medical knowledge of participants and improving the likelihood of their 
subsequent licensing (and thus better utilization in a sparse health 
manpower pool) . 



Impact of the Curso de Perfeccionamiento: 
An Audit of the Effectiveness of the 
Physician Retraining Program at th^e 
University of Puerto Rico 

FINAL REPORT FOR CURSOS 1 AND 2 

INTRODUCTION AND OVERVIEW 

In 1969 5 the Medical School of the University of Puerto Rico (UPR) 
entered into a contract with the National Center for Health Services 
Research and. Development (HSRD), Public Health Service, U. S. Department 
of Health, Education, and Welfare to plan and conduct a retraining 
program for physicians vho had failedj on one or more attempts, to pass 
the licensing examinations of the Puerto Rico Board of Medical Examiners, 
(Such physicians were found to "be, in every case, graduates of foreign 
medical schools.) Educational Testing Service (ETS) vas retained by the 
Medical School, under sub-contract, for a variety of support activi- 
ties which fall into two categories: (l) services to the administrative 
staff and faculty that might facilitate the planning, selection of 
physicians for retraining, or the effective conduct of the retraining 
program (including improving opportunities for internal evaluation and 
refinement of the activity); and (2) ^'third-party" evaluation of the im- 
pact of the program. 

The purpose of the present report is not to inventory the services 
under the first category of support activities by ETS. (That has been 
done, in part, by a previous formal report by ETS dated June 30 5 1970 , 
and filed with the HSRD Project Officer and the UPR Project Director; and 
also in great detail in the periodic progress reports of the UPR Project 
Director.) Rather, the present report is a formal presentation of the 
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accumulated eA^idence to date of the quality and impact of the federally- 
supported, retraining activity, covering the evr^luation of the first 
and second offerings of the special training course. 

Two training programs of approximately 6 months duration, labeled 
the ''Curso de Perfeccionamiento*' (or enrichment course) , were offered 
in succession: the first over the period from July 1970 to December 1970, 
and the second from March 1971 to August 1971. (A third "Curso," 
supported now within the regular budget of the Medical School, has just 
been completed, and its evaluation will be the subject of a subse- 
quent report.) Our purpose is to summarize the naturally-available 
or specifically-collected evidence as to (a) the availability of candi- 
dates for retraining, (b) the acceptability of the rather strenuous re- 
training program designed for them, (c) the program's impact in terms 
of measurable gains in medical knowledge, and (d) in terms of subse- 
quent success in obtaining a license to practice. (Performance upon entering 
practice is the subject of a separate, complex, and yet imcomplete inquiry 
by ETS and UPR faculty and staff, and is not reported at this time.) 

The Need and the "Market^' 

It had been observed by the initiators of the retraining program at UPR 
that there resided in Puerto Rico a large number of graduates o£ foreign 
medical schools who appeared willing or anxious to practice on the Island, but 
who had failed to attain licensure. It was also observed that the shortage 
of practicing physicians in rural areas of Puerto Rico was particularly acute. 

The initial examination of the records o£ the Puerto Rico Board of 
Medical Examiners, covering a ten year period through April 1969, 



revealed the names of 220 physicians who had nevei' achieTOd licensure-- 
and vhohad, on from one to fourteen attempts, failed the licensing 
examinations. The September 1969 licensure administration added 
another hi physicians in this category. Thus, on the eve of the first 
Curso, there appeared to be available a pool of 26l medical school grad- 
uates interested in practicing in Puerto Rico but lonable to obtain a 
license for practice. 

An initial activity was the construction of a biographical and 
training preference questionnaire for this target group of physicians. 
This was mailed to ^.he group of 26l in December 1969- Natural shrink- 
age — physicians moving and leaving no forward rg address, death or . 
retirement, or licensing via another route — brought the available pool 
down ta217. By the end of February^ 139 physicians had returned question- 
naires — as had 105 of their spouses. 

The results of this survey were included in a brief report of April 
10, 1970, and in a detailed report dated "Spring 1970," filed with the 
UPR Project Director and the HSRD Project Officer. Highlights for the 
present purpose, however, may be summarized very briefly. 

First, all but eight of the 139 responding physicians stated an 
interest in receiving:; retraining. Further, almost three-fourths were 
currently residents in Puerto Rico, and more than half expressed a posi- 
tive interest in practicing in Puerto Rico (with many- not responding to 
that question) . Almost all stated an intent to try for licensur.e 
again. The most frequent preference for later activity was practice or 
service in .a government hospital. Very few expressed unwillingness to 
serve a special assignment in a critical health care need area in Puerto 
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Rico, and their spouses, as determined from the separate questionnaire, 
supported this interest in retraining. 

Second ^this portion of the pool of unlicensed physicians revealed 
these characteristics: substantial experience (the median number of 
years of previous medical practice elsewhere was 10); most frequent 
practice in general medicine, internal medicine, surgery, or obstetrics/ 
gynecology; median age, HH; origin about equally divided between Cuba, 
the Dominican Republic, and Puerto Rico (the latter group had completed 
medical school in other countries, principally Spain); and, origin 
principally from comfortably well-to-do families living in cities of 
50,000 or more. 

In sum: there seemed indeed to be an available pool of physicians 
for . retraining. A substantial niomber of that pool expressed interest in 
the retraining opportunity, were it to be presented, and an inclination 
to practice in Puerto Rico,,, if retraining contributed to success in 
attaining a license. 

Success in Fill j ng the Curso, and Subsequent Licensing History 

Stating interest in a retraining opportunity is one thing; taking on 
a strenuous six-month program, with consequent disruption of other activ- 
ity, and physical, if temporary, move to San Juan necessitated in most 
cases, is another thing. 

The decision to proceed with the first Curso was confirmed 
April 1970,- with that program to begin almost Immediately in July 
1970. The staff was able to select and enroll their self-established 
goal of 50 physicians. Of that group, '1+7 proceeded through the Curso 
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(one died and two withdrew) and hk sat for the Medical Boards in December. 
(The three who did not take the examinations were resident aliens, who 
had not lived in Puerto Rico for a sufficiently long period to qualify to 
take the examination. ) Their performance on the Licensing Exam, by parts , 
was: attempted complete Part 1 (Basic Sciences), 38, with 15 passing; 
attempted complete Part 2 (Clinical), 38 with 28 passing. Requirements for 
licensure of U.S. citizens and foreign nationals vary, however. At the 
end of the licensing board action, and in accordance with the differential 
requirements, 33 physicians from the first Curso received some form of 
license. Of those who did not, three of the four who attempted the exam- 
ination did obtair^ a license on the following licensure examination in 
March 1971, including 2 of the 6 physicians receiving Provisional license 
earlier who obtained permanent licensure on the March 1971 examination. 
The general flow of the physicians through the first Curso and the two 
subsequent examinations is given in Figure 1 on the following page. 

The second Curso, begun March 1971 5 attracted and enrolled 3k 
physicians 5 of whom 7 were from the first Curso but whose first post- 
Curso licensing attenpt had bLiou urisuccessful . For this group, 32 of 
the physicians completed the Curso and took the Boards, with I8 approved 
for either a egular or a Special license. Another 5 received a Provi- 
sional license. Of the 1^ participants who either did not pass the first 
post-Curso licensing or received a Provisional license, 5 attempted the 
subsequent licensing examination, ai-^d 2 of them were approved for practice 
with one or another kind of license. The general flow of the physicians 
through the second Curso and the two subsequent examinations is shown in 
Figure 2. 
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The most direct and obvious implication of this history is that the Cursos, 
offered for six months each year (approximately) ?. have been sufficiently attrac- 
tive that from 3^ to 50 physicians have been selected and induced to undertake 
the retraining opportunity. 

A second implication is that there are now practicing (or available 
for practice). 58 additional physicians who have gone through the Curso. 
(One from Curso 1 who failed the Boards following that Curso was sub- 
sequently reexamined and approved.) This is not particularly good 
evidance of impact of Curso, however. One needs to know what the typical 
experience is for physicians retaking the State Boards, as a gauge for 
examining the similar experience for the Curso physicians; further, one 
needs to calculate this experience in terms of number of previous failures 
on the Boards. Beyond this, it would be desirable to break down the 
group into U.S. citizens or aliens, and consequent Board requirements and 
type of licensing options, but at this point it becomes quickly apparent 
that considering all the data cells — pass vs. fail, number of previous 
attempts (which also must be subdivided, as many physicians sit first 
for one part, then another or all of the examinations), citizenship, and 
kind of license — the total of 8I Curso 1 and 2 participants provides 
too small a number for meaningful comparisons. 

As an attempt to focus on the most critical aspects, we have examined 
the numbers of physicians passing or failing each part (basic medical 
science vs. clinical areas), and have, from that record, computed a proba- 
bility of passing the post-Curso Board. Taking the two parts of the 
Boards and the two Cursos separately: 41 physicians in Curso 1 took Part 
1, (Medical Science area) with 18 or 44% passing; 39 phy-sicians took Part 2, 
(Clinical area) with 24 or 62% passing. For Curso 2: 30 physicians took 
^ Part 1, with 9 or 30% passing, and 31 took Part 2, with 20 or 65% passing. 

ERIC 
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Similar data collected for "physicians in general" who graduated 
from foreign medical schools are presented together with that for the 
Curso participants in Table 1. This takes into account the number of 
times specific parts of the Licensing Examination had previously been 
taken. With the exception of those physicians taking Part I of the 
Examination for the first time, Curso participants compare quite favorably 
with "physicians in general." (It should be recalled that non-citizens 
are not required to take Part 1 of the Licensing Examinations.-) 

The Experiment a l Assessment of Impac t of the Curso 
Experimental Variables : ■ 

An early effort in the preparatory activities of the Medical School 
and of ETS vas the development of twelve tests of medical knowledge, a 
Spanish language scholastic aptitude test, and a test of ability to 
understand scientific material presented in written English. The twelve 
medical knowledge tests were developed, through standard ETS procedures, 
using faculty of the Medical School 'to establish specifications and write 
items (after training). Technical and editorial review reduced the number 
of items to ^0 for each medical knowledge test. Topical content of thesa 
tests is: in the basic medical sciences — anatomy, physiology, pharmacology, 
microbiology, biochemistry, and pathology; and in the clinical areas — 
general medicine, pediatrics, psychiatry, commuiTity health, surgery, and 
obstetrics/gynecology . ^ 



No formal analysis of the technical reliability of these tests was made. 
Standard item analyses were' conducted, however, and reveal a good range of 
item difficulty and a high degree of internal consistency. 
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■ TABLE 1 

PROPORTION OF PHYSICIMS APPROVED, BY SUB-EXAMINATION, ON FIRST POST-CURSO 
LICENSING EXAM FOR CURSO 1 AND CURSO 2 PARTICIPANTS 



Part I 



Curso Number Base Rate** 

2 Combined 



# o£ Times 
Licensing 
Exam Was 
Taken 
Pre viously 

0 9/25=. 36 2/14=. 14 11/39=. 28 Cp<.001)* .63 

1 6/8=. 75 4/7=. 57 10/15=. 67 (p<.10)* .44 

2 1/3=. 33 2/4=. 50 3/7=. 43 NS* .46 

3 or more 2/5=. 40 1/5=. 20 3/10=. 30 NS* .24 
TOTAL 18/41=. 44 9/30=. 30 27/71=. 38 .55 

Part II 

# of Times 
Licensing 
Exam Was 
Taken 

Previously 

0 6/8=. 75 — - 6/8=. 75 NS* .59 

1 12/21=. 57 3/5=. 60 15/26=. 58 NS* .52 

2 2/3=. 67 11/14=. 79 13/17=. 76 (p<.005)* .42 

3 or more 4/7=. 57 6/12=. 50 10/19=. 53 (p<.10)* .34 
TOTAL , 24/39=. 62 20/ 31=. 65 44/ 70=. 63 .54 

* Test of significant difference from base rate proportion, using a one- 
tailed binomial exax:t test. 

** Base rate established for all foreign trained physicians taking licen- 
sure examinations over a period from March 1959, through April 1972. 
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The general design of the inquiry, then, was to administer the four- 
teen tests to Curso applicants just before the Curso began; then, in the 
final veek of the Curso, to repeat the twelve medical knowledge tests, . 
toward determining if significant gains had occurred. Also relevant, 
of course, are any "grades" or evaluations made by teaching faculty in 
the progress of the Curso. 

For each Curso, the .ullowing potential matrix of variables becomes 
available: 

1. Scores (or averages) on each of nine pre-Curso licensing 
examinations (the eight tests previously described, plus 
a "practical examination");, 

2. Scores on the twelve pre-Curso medical knowledge tests, the 
scholastic aptitude test (yielding a verbal, quantitative, 
and total score), and the English comprehension test; 

3. One or more instructor-derived evaluations of performance 
within the Curso; 

k. Scores on the twelve jSost-Curso medical knowledge teste; 

5. Scores on the nine post-Curso' licensing examinations. 

In addition, of course, other variables can be derived from summa- 
tions of scores — for example^, the nine licensing examinations can be 
translated into a single pass/fail score — or, from differences in 
pairs of variables — for example medical knowledge or licensing reex- 
amination gain scores. 

Thus , there exists a minimum of U5 different variables on each 
Curso group,^ One statistical hazard of using all — even if in sub-sets 
for different purposes — is, of course, that we have more variables than 

ERIC 
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people. Another is that there are built-in unavoidable interrelation- 
ships among the test variables. In the series of analyses conducted, 
ultimately the relationship of every variable to every other variable vas 
determined, under the strategy that thorough search for tentative trends 
might be more useful than more general analyses of less speculative 
summations. Data from the complete correlation matrices are presented 
in the appendices, and their most important implications may be summar- 
ized belov. It should be noted that these implications are not results 
of rigorous statistical analyses, but rather suggestions from the avail- 
able data. 

1. With regard to selec bion : the preliminary data analyses indi- 
cated that there are sufficient relationships between the pre- 
Curso versus the post-Curso measures to permit selection of (a) 
those most likely to do veil on separate Medical Knowledge 
Tests administered after the Curso; and (b) those most likely 
to do well in the Curso as measui^ed by quizzes administered 

in the Curso; or (c) those most likely to achieve licensure, 
or to achieve a passing score on the individual licensing 
examinations (with the exception of the Obstetrics licensing 
exam — see Appendix A and Appendix B). 

2. Although the relationships are not high, those who improved 
their score the most on the readministration of the Medical 
Knowledge Test were those with the lowest scores on the pre-test 
(this group of course has more room for improvement). Those 
most likely to obtain the higher scores on the post-test — 

or on the licensing exams — were, in general, those with the 



-13- 



higher scores on the pre-test battery (a typical finding in a 
test-retest situation ) . 

With regard to effectiveness of the Curso: first, there is 
some evidence, from the analysis of those who improved the most 
between pre-test and post-test, that the Curso was most effec- 
tive with the lover ability or poorer performing physicians. 
There is also a suggestion in the data. that the more pragmatic 
(as opposed to those oriented to fact and theory) profited more 
from instruction (i-e.^those who do well in' the clinical area 
as opposed to medical science area prior to the Curso, tend to 
improve more in both areas ) . 

In general J however, the measures used during the Ciorso seem 
reasonably well related to the content of the Medical Knowledge 
Tests, both pre-Curso and post-Curse, vhich was a somewhat 
separate but thoughtful specification of what a physician should 
know, 

The measures used during the Curso (reflected in the quizzes) 
do not seem to be significantly related to passing versus 
failing licensing. They are, however, related to a number of 
the separate licensing areas. It is considerably distressing 
that the best outside criterion — passing the medical boards — 
has a low relationship to the midterm tests, but rather clearly 
no relationship to the a21~quiz average or number of quizzes 
passed. The suggestion is that whatever the students were eval- 
uated on after instruction in the various instructional blocks 
is not related to passing licensure exams. 
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6. There is a high positive relationship between the Verbal Apti- 
tude, English Readings and Total Medical Knowledge pretest 
score on one hand and the average on the Curso quizzes ( e.g., 
the predictor variables cited can be used to identify with a 
high degree of accuracy those who will perforin well on criteria 
invoked by the Curso instructors) . 

7. Of larger import: Scores on the Medical Knowledge Pretests are 
significantly related to passing licensure (this relationship is 
developed much more rigorously in a following section). 

8. Taking the separate Medical Knowledge Pretest scores, all but 
one (Community Health) have negative relationships to the Total 
Gain score (a score representing a simple sum of all Post-Pre 
differences). The scores on Physiology, Pathology, General 
Medicine, Pediatrics, and Psychiatry in particular seem to have 
substantial negative relationship to the Total Gain score. In 
other words, those who improved the most from pre-test to post- 
test appear to be those with higher scores cn Community Health 
and lower scores on the five other areas in particular. 

9. Taking the separate Medical Knowledge post-test scores, most have 
low but positive relationship to the Total Gain score, as would 
be expected (e.g., those scoring high on the separate post- 
tests tend to make higher Total Gain scores). However, Community 
Health again appears to stand out in some special way against the 
Total Gain score: there is a high relationship between Community 
Health post-test score and Total Gain score. . 

What these data mean, with regard to the Community Health test, 

O 
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is uncertain. The content of that test seems to be more judgmenta 
and problem solving, and less factual in nature than the other 
tests. It may be performing a subtle role in identifying a 
personal trait or personality style that is associated with im- 
provement in medical knowledge (through the Curso experience) 
generally, (With positive relationship for both pre-test and 
post-test Community Health vs , Total Gain, it is not plausible 
to say the Total Gain accrued because of a good coverage in the 
' Curso of the Community Health area,) 

The Experimental Design for Study of Curso Impact 
Some of the interpretations from the data mass just cited have impli- 
cations for problems other than assessment of impact of the Curso, which 
is, of course, the basic objective of this report. Toward that basic 
objective, we have selected, as the most frequently available^ consistent, 
and reliable data frame for that purpose (l) that deriving from the 
before/after administration of the medical knowledge tests, and (2) that 
deriving from the examination of scores on the pre-Curso vs , post-Curso 
licensing examinations. 

Using these two sets of before-after measures, two important ques- 
tions must be raised. The first — as two Curso groups are involved, and 
as these could differ in initial level of promise (and further, as modifi- 
cations in staff, schedule, and curriculum were made after Curso l): Are 
the two groups of physicians (Curso 1 and Curso 2) similar or different 
in (a) level of promise as revealed by pre-Ciorso variables, or in (b) 
pre-Ciirso/post-Curso gains? The answer to this .question tells us 
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(a) where the two groups may be cOiflbined (to yfeld a larger number for 
and greater reliability to the analyses), and/or (b) if there were im- 
portant differences (and if so, their, probable origin — in student or 
in course characteristics) in impact of the two Cursos. 

The second important question, once the prior question is answered, 

is: Are there significant evidences of gains between the time the Curso 

2 

began and the time it concluded? 

The statistical procedures employed, and the results, are presented 
in the next section of this report. 



An important limitation of the design, imposed for practical and cost 
reasons, is the absence of a "control group." Any gains found may be 
those that would normally occur from self-study, the experience of prior 
testing, the tendency for scores to be error-infested and regression 
toward the mean "true" score on second testing, etc* 
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ANALYSES AND RESULTS 
As specified in the prev:?.ous section, the data collected for both 
Curso groups lend themselves quite naturally to grouping into five 
variable sets. These variable sets can be labeled as follows: (1) 
Pre-Curso licensing scores; (2) Pre-Curso testing variables (including 
the two aptitude measures, English reading, as well as the twelve UPR 
medical knowledge tests); (3) Course scores (the various measures taken 
on Curso participants during the Curso proper, measures which differ 
substantially in content from Curso 1 to Curso 2); (4) Post-Curso Testing 
Variables (the twelve medical knowledge tests); and (5) Post-Curso 
Licensing Variables. Raw score means and standard deviations for these 
five variable sets as well as tables showing intercorrelations between 
and within variable sets have been previously discussed and are given in 
Appendices A and B; Appendix A showing the results for Curso 1 and Appendix 
B showing the results for Curso 2. It should be noted at this point, in 
relation to the statistics reported in Appendices A and B, that in neither 
Curso were all measures available for all Curso participants. For descriptive 
purposes, this problem of '^missing data" has been approached by using all 
available data to compute the various descriptive statistics. For this 
reason, the number of cases contributing to a given statistic will vary. 
For means and standard deviations, specific numbers contributing to each 
statistic are given; however, in the intercorr elation tables only the 
minimum and maximum values of N are given. I^ile the careful reader will 
certainly find interesting and suggestive patterns of correlation presented 
in these two appendices, some of which have been mentioned previously, we 
shall not pursue at this point any further interpretation of such patterns. 
Such interpretation would be clouded as previously noted by (1) the "missing 
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values" problem (which in some cases is quite marked) , and (2) the 
extremely large number of variables relative to the sample size. Both 
of these problem areas serve to reduce the stability of patterns in the 
findings which were discussed in some detail in a previous section. 

Differences Between Curso Groups 

Before asking the question of whether or not the Cprso students made 
significant gains, a natural prior question needs attention. Specifically, 
do the groups of the two Cursos differ in terms of logically related sets 
of variables common to the two Cursos? The variable sets that suggest 
themselves for grouping are: (1) ability (cognitive) variables as measured 
by the Verbal and Mathematics Tests and the English Reading Test; (2) 
medical knowledge prior to the -Curso, as measured by the pretesting on 
the 12 Medical Knowledge Tests; (3) medical knowledge after the Curso, 
as measured by the post- testing on the 12 Medical Knowledge Tests; (4) 
most recent Licensing Sub-*Test scores prior to the Curso; and (5) 
Licensing Sub-Test scores on completion of the Curso. Since in all 5 
insJtances we are dealing with sets of variables, an appropriate statis- 
tic is the Multivariate Analysis of Variance (MANOVA) . MANOVA is de- 
scribed in some detail by Morrison (1967) and Anderson (1958). Loosely 
speaking MANOVA allows the researcher to test differences between 2 or 
more groups on an entire set of variables, simultaneously. The par- 
ticular program used for the analyses reported below is that developed 
and described by Clyde, Cramer and Sherin (1965). It should be noted 
that the means and standard deviations presented in this section will not 
necessarily be the same as those given in Appendices A and B. The reasons 
for the discrepancies is that MANOVA requires no "missing data" for any 
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case on any of the variables; thus those Curso participants, for vhom 
scores are not available on as fev as one of the variables of a partic- 
ular analysis 3 must be excluded from that analysis. Further, the 
participants of Curso 2 included 7 participants of Curso 1 vho failed 
the Medical Licensing exam following Curso 1 and subsequently "reenrolled" 
in the Curso de Perf eccionamiento. While these "repeaters" can be rea- 
sonably included in the results for Curso 1, they may represent a some- 
what different group than those other participants in Curso 2. For this 
reasorj, they have been excluded from any analyses in which their exclu- 
sion has led to markedly different results (they have been included in 
the computations of descriptive statistics in Appendix B). 

Means and standard deviations of the ability variables by Curso 
group are given in Table 2, While these data indicate consistently 
larger means and standard deviations for the first Curso participants, 

TABLE 2 

mM3 AND STMDAED DEVIATIONS FOR ABILITY VARIABLES 



Test 

Curso English 

Number Verbal Math * Re'ading 

1 Mean 25.6 9-9 12.0 
N=i+i+ S.D. 11.3 3.9 7.^ 

2 Mean 2k. k 8.8 11.3 
N=:33 S.D. 10,1 3.h 5.8 
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those differences do not approach statistical significance, either singly 
(by Univariate Analysis of Variance) or collectively (by MMOVA) . There 
is, thus, no indication that either of the two Curso groups started 
training with greater ability (Verbal or Mathematical) or vith an advan- 
tage in English reading. 

Means and .standard deviations of the pre-testing with the 12 Medical 
Knowledge Tests are given in Table 3 . In general the second Curso group 
has slightly larger mean scores and slightly smaller variability; however, 
the statistical test indicates that the groups do not differ significantly 
on this set .of variables when taken collectively. The Curso difference 
on the single variable Microbiology (Univariate Analysis of Variance) 
is significant at the .05 level (F = 3.996). In other words, there is 
no indication that either of the two Curso groups started training with 
an advantage in medical knowledge with the possible exception of Micro- 
biology. _ ., 

In examining the full set of most recent pre-Curso Licensing exam- 
ination scores (8 areas of medical knowledge, excluding the "practical"), 
there was no statistically significant difference between Curso groups 
when the variables were analyzed collectively as a set. For the individual 
comparisons of differences between Curso groups, using each Licensing 
area separately, the groups differed significantly (p<.05) only in the 
area of Tropical Diseases. Unfortunately, only 15 members of the Curso 1 
and 15 members of Curso 2 had previously taken all eight licensing exam- 
inations prior to the Curso, a fact which seriously hampers the examina- 
tion of Curso differences. Due to lack of U.S. citizen status, many of the 
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Curso participants had taken only the Clinical Area Licensing Tests 

(for purposes of obtaining a "Special License") , Due to the reduction 

of sample size imposed on the analysis when the full set of 8 Licensing 

scores was attempted, an additional analysis was performed for the 

Clinical Area Licensing test scores only. The means and standard 

deviations on these examinations for tl.a clinical area are given in 

Table 4. As can be seen in Table 4, Curso 2 participants appear to have 

substantially higher mean scores in the Pre-Curso Clinical Licensing 

areas of "Surgery" and "Tropical Diseases." The statistical analyses 

support this intuitive analysis. The Curso groups differ significantly 

(F = 3.85; df = 4, 67; p<.01) on the set of variables. Examining 

differences on the individual licensing areas shows that the differences 

are, in fact, statistically significant in the area of "Surgery" (F = 8.431; 

df 1, 70; p<.01) and "Tropical Diseases" (F = 6.904; if = 1, 70; p<.05). 

A note of caution should be stressed at this point, however. Licensing 

prior to Curso 1 consisted of a cons7lderabIy less structured and standardized ^ 

procedure than that after Curso 1 ( and immediately prior to Curso 2) . It 

is therefore quite possible that the "most recent" Licensing Examinatior 

scores are not directly comparable, either within or between Curso groups. 

The differences, while significant in a statistical sense, may reflect only 

differences in "most recent" examination taken rather than actual differences 

in Curso participants' medical knowledge. 

Means and standard deviations of the post-testing with the 12 Medical 
Knowledge Tp.sts are given in Table 5 . These data suggest a pattern; 
specif ically , '"of the two groups which began the Curso with no noteworthy 
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difference in ability or medical knowledge, the second gro up ended the 
Curso with an advantage in medical knowledge over the first group on all 
6 of the clinical Medical Knowledge Tests (they were higher on only 2 of 
the Basic Medical Science Tests). MANOVA analyses substantiated the 
hypothesis of Curso group differences, as shown in Table 6. While the 
two Curso groups differed significantly (a = .05) on but one of the 12 
tests (Biochemistry) when considering them separately, the group dif- 
ferences on' the entire Battery of Tests was significant at the .001 level. 
Separate MANOVA analysis for only tile Clinical Area Tests showed no 
significant group differences; thus the post-Curso differences between 
groups can be attributed to the subset of Basic Medical Science Tests. 
In other words {^^^o groups began the Curso as more-or-less equivalent 
on ability and medical knowledge, but these -same two groups ended the 
Curso with statistically significant differences on medical knowledge. 

This suggests, since no prior Curso group differences existed on these 
tests, that the amount of gain on medical knowledge is different for the two 
Cursos. This hypothesis is borne out qualitatively by the graphs of pre-test- 
ing and post-testing for each Ciorso group on each of the 12 Medical Knovrl- 
edge Tests (Figures 3 and 4). The graphs suggest an interaction between 
group and time of testing ( i . e. > different ial gain). To test this hypothesis 
quantitatively, MANOVA was applied to the gain scores for the 12 tests 
(cf.., Cole & Grizzle, I966) . The results of the analysis are given in 
Table 7. Considering the tests individually, the gro'L^ps differed in 
gain significantly (a =: .05) on three tests. Biochemistry, Anatomy, and 
Physiology; and in terms of the entire battery of tests, the groups 
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TABLE 6 

RESULTS OF MULTIVARIATE AND Ul^VARIATE ANALYSES OF VARIANCE FOR CURSO 
DIFFERENCES ON MEDICAL KNOWLEDGE TESTS (POST -TESTING) 



Multivariate Test^ 



Degrees of 
Freedom for 
Hypothesis 

12 



Degrees of 
Freedom for 
Error 

65 



4.260 



p<.001 



Univariate Tests 
Anatomy 
Physiology 
Pharmacology 
Microbiology 
Biochemistry 
Pathology 
Surgery 
Obstetrics 
General Medicine 
Public Health 
Pediatrics 
Psychiatry 



76 
76 
76 
76 
76 
76 
76 
76 
76 
76 
76 
76 



3.304 
.446 
.889 
.053 
21.710 
.373 
.180 

2.190 
.653 

2.556 
.099 
.807 



p<.l 



p<.001 



■^'est of significance using Wilks' lambda criterion. 
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TABLE 7 



RESULTS OF MULTIVARIATE MB bl\fIVAEIATE ANALYSES OF VARIAI\[CE FOR 
CURSO DIFFERENCES IN GAIN ON THE 12 MEDICAL KNOWLEDGE TESTS 



. Degrees of Degrees of 

Freedom for Freedom for 

Hypothesis Error F 

Multivariate Testl 12 6^1 ^.636 p<.001 
Univariate Tests 

Anatomy 1 75 i+.126 p<.05 

Physiology 1 75 i|.8l5 P<.05 

Pharmacology 1 75 I.38H 

Microbiology 1 75 2.672 

Biochemistry 1 75 2H.85H p<.001 

Pathology 1 75 .290 

Surgery 1 75 .32^ 

Obstetrics 1 75 ,985 

General Medicine 1 75 .87I 

Public Health 1 75 I.O56 

Pediatrics 1 75 .2^1 

Psychiatry 1 75 .007 



1 

Test of significance using Wilks' lambda criterion. 
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differed, in terms of gain, at the ,001 level of significance. That is 
to say, there vas a statistically significant difference in the patterns 
of change in medical knowledge between the tvo Ciorso groups. The nature 
of this difference has been previously described and is shown in Figures 
3 and h. S\aimnat ively , Curso grou-p tvo gained significantly more on 
Biochemistry than did Cur so group one, vhile Curso group one gained 
significantly more on Anatomy and Physiology than did Curso group tvo . 
Additional analyses on the subset of 6 Clinical Area Tests shoved no sig- 
nificant Curso group differences either collectively or singly on these 
tests; thus, the differential gain is specifically attributable to the 
Medical Science Test areas. 

Interpretation of these statistically significant differences is ex- 
tremely di'fficult. Differential gains may be due to differences in actual 
conduct of the Curso (different instructors, different educational techniques, 
different vork and study schedules, or even to physical differences in Curso 
facilities) or to differential selection procedures leading to differences in 
participants (to rule out these last tvo possibilities^, the selection proce- 
dure vas maintained as more-or-less the same and previous analyses have sug- 
gested few pre-Curso differences for the tvo groups). More sophisticated sta- 
tistical approaches vhich mathema"^ ically control for existing group differ- 
ences (Covariance Analysis) vere not attempted due to small group sizes rela- 
tive to the large number of variables vhich vould need to be considered. 

Table 8 shovs post-Curso group means and standard deviations on the 9 
Licensing Examination Sub-tests. Some rather marked differences betveen 
the tvo Curso groups are evident from the table. Group differences vere 
tested quantitatively by MAN0VA5 the results of this test are given in 
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Table 9. The analyses shbv the tvo groups differing on the Battery of 
Licensing Sub-Tests at the .001 level of significance. Additionally, 
significant (a = .05) Curso group differences exist for the individual 
sub-test comparisons on Anatomy, Surgery, and Obstetrics. By reference 
to Table 8, it is seen that these differences are attributable to the 
higher performance of Curso group 1 on the Anatomy and Surgery Sub-Tests 
an'^ the higher performance of Curso group 2 on the Obstetrics Sub-Test. 
Reason for these differences may be attributable to pre-Curso differences, 
in some cases, or to differential Curso benefit. It should be kept in 
mind, however, that the tvo Curso groups did not take the same licensing 
examinations following the course, although both groups took Post-Curso 
Board Examination after the inception of a new series of objective-style 
tests (where 'conscious effort and technical assistance from ETS in a 
separate project--directed toward ^^standardizing'^ this procedure). 

In any event, the tvo groups, which began the Curso more-or-less equal 
in ability and medical knowledge, performed differentially on the Battery 
of Licensing Tests administered after the Cursos, For this reason, an 
analysis of differential gains on licensing exa^ninations was performed. 
Due to limited availability of pre-Curso licensing examination results in 
the Medical Science area for both Curso groups (and the great variation, 
presumably, in pre-Curso licensing exams), this analysis was performed 
only for the k Clinical Area licensing examinations. The differential gain 
on these licensing examinations is shown in Figures. Very marked inter- 
actions (reflecting differential gain) are evidenced in the areas of 
Surgery and Obstetrics, the gain being greater for Curso 1 participants 
in the former and for the Curso 2 participants in the latter. Further there 
is some evidence of greater gain for the Curso 1 group in the area of 
Tropical Diseases, The statistical analysis of these gains supports the 
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TABLE 9 

RESULTS OF MULTIVARIATE AND UNIVARIATE ANALYSES OF VARIANCE FOR 
CURSO DIFFERENCE ON LICENSING SUB-TESTS (POST-CURSO TESTING) 



Degrees of Degrees of 
Freedom for Freedom for 
RVPothesis Error 



F 



Multivariate Test 



51 



p<.001 



Univariate Tests 
Anatomy 
Physiology 
Pharmacology 
Pathology 
Siirgery 
Obstetrics 
General Medicine 
Tropical Diseases 
Practical 



1 
1 
1 
1 
1 
1 
1 
1 
1 



59 
59 
59 
59 
59 
59 
59 
59 
59 



13.521 
1.5^0 
.282 
1.096 
17.1^6 
kl.hn 
.533 
.109 
1.135 



p<.001 



p<.001 
p<.001 



Test of significance using Wilks' lambda criterion. 
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GENERAL MEDICINE TROPICAL DISEASES 



Figiire 5' Gains on licensing examinations 
(clinical area) for Cursos 1 and 2. 
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TABLE 10 

RESULTS OF MULTIVARIATE AND UNIVARIATE ANALYSES 
OF VARIAI^CE FOR CURSO DIFFERENCES IN GAIN 
OH THE h CLINICAI. AREA LICENSING EXAMINATIONS 



Multivariate Test 

Univariate Tests 
Surgery 
Obstetrics 
General Medicine 
Tropical Diseases 



Degrees of 
Freedom for 
Hypothesis 



1 
1 
1 
1 



Degrees of 
Freedom for 
Error 

58 



61 
61 
61 
61 



lk.k09 

26.lQk 
12.657 
0.010 
6.598 



p<.001 

p<.001 
p<.001 

p<.05 



Test of significance using Wilks ^ lambda criterion, 



observations derived from the figure, and these results are given in Table 
10. It can be seen from Table 10 that not only is the overall pattern of 
gain significantly different in a statistical- sense between the two Ciorso 
groups for the set of licensing examinations , but also the gain differences 
are statistically significant for three of the four tests when considering 
them individually. The direction of differential gain is obvious from 
Figure 5. 

The results of our analyses for differential gain appear on the surface 
to be contradictory. In terms of the 12 UPR Medical Knowledge Tests ^ 
differential gain was strictly attributable to the Medical Science area sub- 
tests, while in terms of Licensing Examinations, there are clear-cut gain 
differences in terms of those examinations in the Clinical Area. It should 
be pointed out, however, that the contradiction may be a fiinction of variables 
not directly related to the Curso but rather to the nature of the two sets of 
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instruments used and to certain restrictions placed on the data by using 
the most generally appropriate statistical tools. The first matter , 
, basically that of non-standardization of the licensing instruments has 
been discussed elsewhere in this report. The second matter has also 
been mentioned but needs some clarification. Specif ically 3 our two 
tests for differences in gain vere not, strictly speaking, performed for 
the exact same people. The test for UPR test gain differences vas for 
those Curso participants for vhich both pre- and post-Curso test scores 
were available. The test for licensing examination gain differences 
was for those Curso participants for which both pre- and post-Curso 
licensure examination scores were available; and the groups so defined 
are not identical (i.e. > some participants for whom UPR test results 
were available did not have available licensure examination results in 
the Clinical area prior to the Curso, others did not taite the licensure 
examination on completion of the Curso for various reasons). Due to 
these possible distortions of the results , one should be cautious in drawing 
too many conclusions from the differential gains . 

Overall Gains in Medical Knowledge for the Two Curso Groups 

The question remains as to the significance of the overall gains 
for the two Cursos on the ].2 Medical Knowledge Tests and for the clinical 
area licensure examination. Since it has been determined that the two Curso 
groups gain di.'f'ferentially in both areas , the appropriate tests of gains 
are within Curso groups. These analyses were, performed , and the results 
are repoi'ted xn Tables 11 through lU. As can be seexi from Tables 11 and 
12 the gains in both groups were highly significant ones on every individual 

ERLC 
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TABLE 11 

MULTIVARIATE AND UNIVAEIATE TESTS FOR SIGNIFICANT GAINS ON 
MEDICAL KNOWLEDGE TESt:'S FOR CURSO GROUP 1 



Multivariate Test 
Univariate Tests 
Anatomy 
Physiology 
Pharmacology 
Microbiology 
Biochemistry 
Pathology 
Surgery 
Obstetrics 
General Medicine 
Public Health 
Pediatrics 
Psychiatry 



Degrees of 
Freedom for 
Hypothesis 

12 

1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 



Degrees of 
Freedom for 
Error 

32 



43 
43 

A3 
43 
43 
43 
43 
43 
43 
43 
43 
43 



17.792 

34.785 
79.324 
18.059 
40.488 
11.454 
43.435 
13.610 
32.762 
91.321 
34.a82 
22.224 
14.509 



p<.001 

p< .001 
p< .001 
p< .001 
p<.001 
p<.005 
p<.001 
p<.001 
p<.001 
p< .001 
p< .001 
p< .001 
p<.001 



Test of significance using Wilks' lambda criterion. 
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TABLE 12 • 

MULTIVARIATE AND UNIVARIATE TESTS FOR SIGNIFICANT GAINS ON 
MEDICAL KNOWLEDGE TESTS FOR CURSO GROUP 2 



Degrees of Decrees of 
Freedom for Freedom for 
Hypothesis Error 



Multivariate Test' 



12. 



21 



19.597 



r)<.001 



Univariate Tests 
Anatomy 
Physiology 
Pharmacology 
Microbiology 
Biochemistry 
Pathology 



Sirr^ery 



Obstetrics 
General Medicine 
Public Health 
Pediatrics 
Psychiatry 



1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 



32 
32 
32 
32 
32 
32 
32 
32 
32 
32 
32 
32 



5.611 

ik.euQ 

5.166 
19.270 
86.606 
29.320 

19.956 
36.867 
57.156 

62.il82 

31.508 
17.085 



P<.05 

p<.001 

P<.05 

p<.001 

p<.001 

p<.001 

p<.001 

p<.001 

p<.001 

p<.001 

p<.001 

p<.001 



Test of significance using Wilks' lambda criterion. 
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TABLE 13 ■ 

MULTIVARIATE AND UNIVARIATE TESTS FOR SIGNIFICANT 
GAINS ON CLINICAL AREA LICENSURE EXAMINATIONS 
FOR CURSO GROUP 1 



Multivariate Test 

Univariate Tests 
Surgery 
Cbstetrics 
General Medicine 
Tropical Diseases 



Degrees of 
Freedom for 
Hypothesis 



1 
1 
1 
1 



Degrees of 
Freedom for 
Error 

28 



31 
31 
31 
31 



50 ,1119 
0.001 
23-I472 
79. 93^^ 



p<.001 

p<-001 
— {. 

p<.001 
p <.001 



Test of significance using Wilks ' Lambda criterion* 



TABLE Ih 

MULTIVARIATE AND UNIVARIATE TESTS FOR SIGNIFICANT 
GAINS ON CLINICAL AREA LICENSURE EXAMINATIONS 
FOR CURSO GROUP 2 



Multivariate Test 

Univariate Tests 
Surgery 
Obstetrics 
General Medicine 
Tropical Diseases 



Degrees of 
Freedom for 
Hypothesis 



1 
1 

1 
1 



Degrees of 
Freedom for 
Error 

27 



31 
31 
31 
31 



16,628 

• 1,267 
28.165 
i+5.822 
27.518 



p<.001 



p<.G01 
p<.001 
p<.001 



Test of significance using Wilks ' lambda criterion. 



test of the UPR Medical Knowledge Battery (and for the Battery as a whole). 
This would tend to indicate an extremely beneficial impact of the Curso on 
both groups in terms of scores on the Medical Knowledge Tests (in both the 
basic medical science areas and in the clinical areas). 

Tables 13 and 1^ indicate that overall gain on the Clinical Area . 
Licensing Examinations was significant, considering the entire set of 
examinations, for both Curso groups. For Curso 1 participants, the gain 
was significant on every individual test area except that of Obstetrics 
(in fact there was a very slight loss in that area). Curso 2 participants 
showed significant gains for all areas except that of Surgery. The 
overall picture is again indicative of a beneficial impact of the Curso 
in terms of Licensure examinations (at least for the Clinical Areas). 

There are problems in interpretation, mentioned before, that some- 
what cloud these indications; and, further, it can be argued that some gain 
is expected on the UPR Medical Knowledge Test Battery since the same 
tests were given at both administrations. In spite of such limitations 
on our findings, the preponderance of the findings strongly support the 
notion that the Curso de Perfeccionamiento ^or both Curso groups) has 
served to increase substantially the medical knowledge of the participants. 

Predictability of Licensure from Pre-Curso Data 

So^.e comment has been made earlier regarding the general lack of single 
variable relationships of p're-Curso variables to the dichotomous variable of 
post-Curso licensure. Two basic approaches to this problem 'are standardly- 
employed, using in both cases the entire set of ^'predic tor^* variables. One 
is the Multiple Regression approach, treating the dichotomous outcome as 
a two-valued variable (e.g., persons obtaining license.- can be assigned a 
score of 1, and those not obtaining license can be assigned a score of O). 
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The topic of multiple regression is treated extensively in Draper and Smith 
(1966) and in most advanced statistics tests. The other approach is that 
of Discriminant Analysis (cf,, Morrison, 1967), which finds some "best" 
dimension, representing a linear combination (a weighted sum) of the 
variables of interest, which maximally discriminates between two groups. 
We have chosen the latter approach for various technical reasons. Specif- 
ically we have used stepwise discriminant analysis, a technique which adds 
variables, one at a time, to. the discriminant function on the basis of a 
predetermined decision rule. The specific program used was the BMD Stepwise 
Multiple Discriminant Analysis program (cf.,. Dixon, 196k). The decision 
rule adopted is a standard one, add that variable, of those remaining, which 
maximally discriminates between the two groups , given any discrimina^j 
power related to those variables ( i f any ) already contributing to the 
discriminant function has been partialled out . Loosely speaking, the 
variable added is the one most discriminating any previously undiscriminated 
differences between the groups. The variables used in our discriminant 
analysis were the pre-Curso scores on the UPR MedicsLl Knowledge Tests and 
the pre-Curso licensure examination scores in the Clinical Area (a total 
of 16 variables). The two groups were (l) those obtaining license on the 
first Board examination following the Curso, and (2) those not obtaining 
license. For purposes of this analysis, Curso 1 and Curso 2 participants 
were combined (both groups were reduced in size' due to "missing data" on 
some variables, and the "repeaters" in Curso 2 were not used). The combi- 
nation of groups seems justified due to the similarities in the two Curso 
groups of the relational patterns among the l6 variables of interest here. 
The resulting groups included a total of 67 Curso participants {kO who 
were licensed and 27 who were not). 
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As a result of the analysis it was determined that optimal discrimi- 
nation between the Licensed and not Licensed group was obtained by using 
only 3 of the l6 variables. All three of these variables were from the 
set of UPR Medical Knowledge tests. They were the tests in the area of 
Anatomy, Surgery, and Psychiatry. The partial F values (representing the 
independent discriminability of the three variables after any discriminability 
of the remaining two has been partialled) are given in Table 15 , 

TABLE 15 

PARTIAL F VALUES FOR THE THREE VARIABLES 
OF THE DISCRIMINANT FimCTION 



Variable 


Weight 


F 


df 


P 


Anatomy 


.123 


2.T8OI 


1,63 


p<. 2 


Siargery 


.160 


3.6966 


1,63 


p<.l 


Psychiatry- 


.116 


3.0001 


1,63 


p<.l 



The partial F values for those variables not included in the discriminant 
function (representing the residual discrimination of these variables after 
the discriminability of all three variables included in the discriminant 
function has been removed) are given in Table 16. From a casual obser- 
vation of these two tables, it is clear that the variables of the discrim- 
inant function have greater discriminability than do those not included (as 
should be the case). The dimension (canonical variable. ) which best 
discriminates the Licensed and not Licensed groups is given, as previously 
stated, by a weighted sum of the scores on the three tests contributing to 
the discriminant function. The weights are given in Table IS above. It 
should be noted that those weights are applied to the "standardized" scores 
(z. scores, computed by subtracting the mean and then dividing by the standard 
deviation) and not the raw test scores. 
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TABLE 16 

PARTIAL F VALUES FOR THE THIRTEEN VARIABLES 
NOT INCLUDED IN THE DISCRIMINANT FUNCTION 

Variable F df 

UPR Medical Knovledge Tests 



Physiology 


.2826 


1,62 


■ P> 


.5 


Pharmacology 


.518U 


1,62 


P> 


.5 


Microbiology 


.0839 


1,62 


P> 


.5 


Biochemistry 




1,62 


P> 


.5 


Pathology 


.0526 


1,62 


P> 


.5 


Obstetrics 


1.2925 


1,62 


P> 


.2 


General Medicine 


I.56U7 


1,62 


P> 


.2 


Public Health 


.2122 


1,62 


P> 


.5 


Pedi atrics 


.0618 


1,62 


P> 


.5 


ng Tests (Clinical Area 


) 








Surgery 


.2077 


1,62 


P> 


.5 


Obstetrics 


.I32U 


1,62 


P> 


.5 


General Medicine 


I.I5HI 


1,62 


P> 


.2 


Tropical Diseases 


1.2100 


1,62 


P> 


.2 
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TABLE 17 

FREQUENCY DISTRIBUTIONS OF TWO GROUPS 
ON CANONICAL VARIABLE 



Frequency for Frequency for 

Value of Group Obtaining Group Not Obtaining 

Canonical Variable License License 



-^.0 to -3.5 1 
-3.5 to -3.0 

-3.0 to -2.5 2 

-2.5 to -2.0 1 

-2.0 to -1.5 • 3 

-1.5 to -1.0 1 

-1.0 to -0.5 k 3 

-0.5 to 0.0 k 5 

0.0 to 0.5 10 3 

0.5 to 1.0 9 3 

1.0 to 1.5 7.1 

1.5 to 2.0 5 
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The frequency distribution of scores for the two groups in terms of this 
"best discriminating" dimension is given in Table 17. From the table, 
it can be seen that the group obtaining license has a higher average 
score than the group which ^ did not obtain license. The group means are 
-.80 for the no License group and .5^ for the group obtaining license. 
The discrimination is reflected in the multivariate F value (corresponding, 
in a loose way, to the univariate F for group differences on the canonical 
variable). The F value obtained vas 9-3^3, vith 3 and 63 degrees of 
freedom, which is significant at p<.001. 

Of greater practica.l significance is the degree to which our post hoc 
classification scheme is successful in determining those who obtain license 
and those who do not. The correct and incorrect classifications, based on 
a cutoff point which is the vc.lue of the canonical variable half the 
distance between the two group means, are given in Table 18. From the table 



TABLE 18 



CORRECT MD INCORRECT CLASSIFICATIONS 
OF CURSO PARTICIPANTS 
BASED ON DISCRIMINANT ANALYSIS 



Actual 
Group 

Membership 



Classified as 
Belonging to Group 



Obtaining 
License 



Not Obtaining 
License 



Obtained 
License 



33 - 



T 



Did Not 

Obtain 

License 



9 



18 



it can be seen that 51 of the 67 (over ^6%) of the Curso participants were 
correctly classified on the basis of only three Pre-Curso Variables . This 
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statistic is not as impressive as it would appear at first glance, for it 
should be remembered that if ve arbitrarily classified a person randomly 
into the licensed group or unlicensed group by the flip of a coin, then 
we would expect to correctly classify half of the persons so assigned 
(also, obviously, if we were to classify everyone into the licensed group 
we would have obtained exactly ko correct classifications out of 6? 
(60^)). V/e can, however, examine our improvement in classification as 
compared to that which we would expect by random assignment. Our propor- 
tional improvement over random assignment is given by (51-33.5)/33.5 or 
.52. Thus we have a 52?^ improvexaent over chance assignment classification. 
The significance of this improvement ovtr chance can be computed by the 
normal approximation to the binomial process we would obtain by random 
assignment (using p=l/2, n=67)- The computed corrected z is H.15 with 
level of significance less than .001. We can 'therefore state with 
considerable confidence that our discriminant analysis classification scheme 
is in fact an improvement over chance classification, and while the 76% 
correct classification rate is less impressive than it could be, it still 
provides an improved selection process. 

It should be noted at this point that classification schemes based on 
such techniques as we have used often depend greatly on the specific group 
for which it is computed. That is to say, since the approach^ uses the 

most discriminating combination of variables based on available data, there 

3 

is room for spurious relationships. Recent empirical studies have shown 
dramatically the instability of "optimal" weights under use with a different 
sample. Some "cross-validation" of this predictor should be undertaken 

3 

See, for example, the work of Dawes of the Oregon Research Institute. 

ERIC 
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prior to using it as a screening tool. Nevertheless, considering the 
relatively small number of variables used in the discriminant function 
compared to the sample size., it is certainly indicated that the UPR 
Medical Knowledge Test Battery (specifically the three tests entering 
our discriminant fimction) could be used effectively in screening those 
applicants, for future Cursos , vho have greatest potential in terms of 
passing the Licensing examination. To be sure, such screening is 
completely unwarranted as long as supply exceeds demand for the Curso 
and as long as funds are available for training all applicants. Errors, 
of one type or another, are always likely with any less-than-perfect 
selection rule. The major errors here are (l) selecting' for training 
individuals who will not benefit (as reflected in their not obtaining 
license), and (2) excluding from training those who could benefit. It is 
not the significance of a predictive eq^uation that is critical in such a 
case, but rather the careful weighting of the costs involved (and we mean 
by costs much more than those measurable in dollars) in making each of 
the two types of inevitable errors. 

In this regard, it should also be pointed out that we considered in 
ovr prediction only the first administration of the Board examination 
following the Curso. It has been mentioned previously that some Curso 
participants who did not obtain license at this time did, in fact, obtain 
license on the following administration of the Board Exams. Further, we 
have not distinguished between Provisional, Special.^ or Regular License 
or between U. S. citizenship status. The careful reader will recognize 
that a type of "residual" analysis may clarify this point. 
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Individual Differences 

Throughout our discussion ve have concentrated on aggregates and 
averages in descrilDing the effects of the Curso. It is quite instructive, 
however, to consider the individual Curso participant to obtain a "feel" 
for the thrust of the data. Obviously, examination of individual profiles 
for all participants on all variables presents a cumbersome task, the 
result of which is such a magnitude of data that one easily loses sight 
of the forest due to all the trees. For this reason we will concentrate 
on those data considered most reliable, the UPR Medical Knowledge Tests, 
and on a specific sub-group of Curso participants, the "repeaters" in 
Curso 2. This group is not chosen arbitrarily, but rather due to the .act 
that measures on the variables are available at three points in time (Pre- 
Curso 1, Post-Curso 1, and Post-Curso 2). The UPR Medical Knowledge Test 
Profiles for 6 of the "repeaters" are given in Appendix C (one "repeater" 
did not take the tests on completion of Curso 2). While it is surely unsound 
to generalize from such a small (and certainly unrepresentative) sample , the 
changes in test profiles from one administration to another reflect, to 
an extent, the broader data base. Scores for a given individual on a given 
test obviously do' not show uniform or even consistent improvement. Where 
test scores actually decline from one administration of a test to the next, 
this probably reflects the unreliability of the test. Perhaps the most 
important observation is that there is a consistent overall raising of the 
test profile following both of the Cursos • 
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CONCLUSIONS 

Considering the necessary limitations of the design, the very impor- 
tant question of relative impact of the Curso can not ^ answered in any 
definitive way. The resulr.s do indicate, however, that regarding Licensing 
Examinations , the Curso participants in general do as well as or better than 
comparable groups of foreign- trained physicians. It is also possible, 
with the data at hand, to speak directly to the question of absolute impact 
of the Curso on its participants and otherwise. 

It was obvious from the early exploratory work that a substantial 
number of potential consumers existed on the Island. The relative ease 
with which the Curso attracted these consumers speaks to the attractiveness 
and credibility of the Curso as a viable instrument which could serve the 
medical needs of the Island and at the same time provide appropriate 
employment tc foreign-trained physicians. 

By all measures available, both Cursos appeared to be successful in 
improving the medical knowledge of the participants. This is overwhelmingly 
obvious from the analyses of Curso gains on both the UPR Medical Knowledge 
Test and on the licensing examination (subject to limitations previously 
specified) . 

In terms of producing sorely needed health manpower personnel in the 
form of licensed physicians, the Cursos were likewise successful. From a 
cost-efficiency point of view, the Cursos should fare quite well relative 
to other means of producing licensed physicians. The impact of this effect 
of the Curso goes considerably beyond the confines of the academic and 
clinical training facilities, extending to the potentially improved health 
care of hundreds of people on the Island. 
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There are now available effective screening instruments to be 
used for future Cursos, should the need occur. These instruments may in fact 
prove to be beneficial tools for evaluation in other areas of physician 
training. 

Finally, the broader impacts of the Curso on medical planning for 
health care delivery systems, potential and untapped health care personnel 
pools, health care education, etc., are too numerous and diverse to 
document here. Suffice it to say that the planning, implementation and 
conduct of the Cursos have proved to be a catalyst for increased innovation 
within the medical education community. These intangible and diffuse 
benefits are most likely the greatest success of the Curso^ in the final, 
analysis . 
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APPENDIX A 



MEANS, STANDARD DEVIATIONS, 
AND qORRELATIONS FOR 
VARIABLE SETS 
CURSb 1 
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TABLE A-1 






MEANS 


AKD STANDARD DEVIATIONS OF 




PRE- 


-CURSO VARIABLES 


(CURSO 1) 








Standard 




V cLxl cLU J-c 


Mean 


V J. CL U J. \JiL 


N 


Verbal Aptitu:l«2 


25. 2U 


11.35 


U3 


Mathematical Aptitude 


9.76 


3.91 


h3 


English Reading 


11.95 


7, ho 


kh 


Anatomy- 


IU.3I+ 


3.91 


hi 


Physiology 


13. 3U 


It. 82 


hi 


Pharmacology 


IT. 26 


k.lk 


hi 


Microbiology 


13.68 


3.71 


hi 


Biochemistry 


16.3!5 


U.60 


he 


Pathology 


15.32 


5.28 


hi 


Siirgery 


11.21 


3.5U 


hi 


Obstetrics 


19.19 


1^.28 


hi 


General Medicine 


15.72 


5.22 


hi 


Public Health 


15. U5 


3.78 


hi 


Pediatrics 


17.87 


h.k3 


hi 


Psychiatry 


15 M 


3.92 


hi 



TABLE A-2 





MEMS MD STANDABD DEVIATIONS 


OF 






PRE-CUESO LICENSING (CURSO l) 




Licensing Test 




Standard 




Meaji 


Deviation 


I 


Anatoircy 


57.29 


21.7i| 


17 


Physiology 


63.76 


10.97 


17 


Pharmacology 


61.9k 


15.30 


17 


Pathology 


62.88 


lk.k5 


17 


Surgery 


55.k6 


15.83 


39 


Obstetrics 


72.63 


11.1k 


38 


General Medicine 


63. k2 


12.70 


38 


Tropical Diseases 


59.89 


13.60 


38 



o 
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TABLE A-3 

MEANS AND STANDARD DEVIATIONS FOR 
POST-CURSO VARIABLES (CURSO l) 



Standard 

Variable Mean Deviation N 

Anatomy 17.51 3.91 , ^5 

Physiology I8.6O ^.30 if 5 

Pharmacology 19-71 3J!-9 ^5 

Microbiology 17-^7 3.85 ^5 

Biochemistry 18. 98 3.97 ^5 

Pathology 20.38 .5-21 

Surgery 13. 09 3-55 ^5 

Obstetrics 22.58 ^.1^ ^5 

General Medicine 21.18 ^1.86 

Public Health 18.98 3.73 ^5 ^ 

Pediatrics 21.67 ^-98 k3 

Psychiatry I8.OO U.02 



A-4 

TABLE A-h 

mmS AND STANDARD DEVIATIONS OF 
COURSE SCORES (CURSO l) 



Standard 

y^^^^'ole Mean Deviation N 

Number of Quizzes Taken 9.28 I.98 I13 

Number of Quizzes Passed 7.37 2.02 ii3 

Midterm 1 78.02 13.88 lii 

Midterm 2 63.65 9-88 k3 

Mean Standard Scoi-e All Quizzes 50.38 5.i|ii i^O 

Pass-Fail O.58 O.5O ii3 

Rating 38.69 lit. 33 1^5 
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TABLE A-5 

MEANS AND STANDARD DEVIATIONS OF 
POST-CURSO LICENSING 



Standard 

Licensing Test Mean Deviation N 

Anatomy 68,8? 10,65 38 

Physiology 72,21 12,17- 39 

Pharmacology 72, 6l 9-78 38 

Pathology 77-03 10,13 39 

Surgery 77.79 7-17 39 

Obstetrics 73.21 7-90 38 

General Medicine 77.82 7.92 38 

Tropical Diseases 80,71 5.62 38 

Practical 75.30 ll,ii6 37 
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APPENDIX B 



^MEMS, STANDARD DEVIATIONS, 
AND CORRELATIONS FOR 
VARIABLE SETS 
CURSO 2 



B-1 





TABLE B-1 








MEANS MD STANDARD DEVIATIONS FOR 
PRE-CURSO VARIABLES (CURSO 2) 




Variable 


Mean 


S"han d ard 
Deviation 


N 


Verbal Aptitude 


2I4.O3 


10.21 


3k 


Mathematical Aptitude 


8.7^ 


3.hk 


3k 


English Reading 


11,27 


5.77 


33 


AnatoiBiy. 


1^.56 


k,02 


3k 


Physiology 


1^.82 


U. 52 


3k 


Pharmacology 


17.27 


k.lk 


3k 


Microbiology 


15.32 


3.39 


3k 


Biochemistry 


15.88 


U.35 


3k 


Pathology 


15.29 


i|.23 


3k 


Surgery . 


11.00 


2.75 


3k 


Obstetrics 


19.65 


I1.21 


3k 


General Medicine 


I7.2I1 


3. kg 


3k 


.Public Health 


15.88 


3.26 


3k 


Pediatrics 


17.53 


il.72 


3k 


Psychiatry 


16.00 


i|.13 


3k 
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TABLE B-2 





MEMS AI^D STAIIDAED DEVT.ATI0I5S FOR 






PRE-CURSO LICENSING 


(CURSO 2) 








Standard 




License Test 


Mean 


Deviation 


N 


Anatonry 


63 


16.37 


15 


Physiology 


6k. 00 


12.39 


15 


Pharmacology 


66.00 


7.1+6 


15 


Pathology 


66. ko 


12. k3 


15 


Surgery 


65'2k 


11.90 


3h 


01)stetrics 




11. k6 


3k 


General Medicine 


63.03 


15.63 


3h 


Tropical Diseases 


6l.9k 


12.23 


3h 
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TABLE B-3 

MEANS AND STANDARD DEVIATIONS FOR 
POST-CURSO VARIABLES (CURSO 2) 



St andard 

Variable Mean Deviation N 

Anatomy 15-88 3.9^ 33 

Physiology ' 17.91 ^.57 33 

Pharmacology l8.82 ^1.88 .33 

Microbiology 17-67 3.69 33 

Biochemistry 23.58 ^.72 * 33 

Pathology 19.6^1 5.^1- 33 

Surgery 13.^2 3.31 33 

Obstetrics 23.95 3.83 33 

General Medicine 22.09 5-03 ' 33 

Public Health 20.21 2.80 33 

Pediatrics 22.03 5.1^ 33 

Psychiatry 18.79 3-55 33 



B-4 



TABLE B-k 

mmh MD STMDARD DEVIATIONS FOR 
COURSE SCORES (CURSO 2) 



Standai'd 

Score Mean Deviation 



Midterm 1 
Midterm 2 



100.17 
91.8k 



13.82 
19.22 
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TABLE B-5 





MEAi^S AND STANDARD DEVIATIONS FOR 
POST-CURSO LICENSING (CURSO 2) 




Licensing Test 


Mean 


Standard 
Deviation 


N 


Pass-Fail 


0.68 


0.1f8 


31 


iuiatoitiy 


56. ill 


17.03 


29 


Physiology- 


75.79 


9.70 


29 


Pharmacology 


71.55 


9.51 


29 


Pathology 


75.17 


8.3k 


29 


Siargeiy 


67.81i , 


10.21 


31 


Obstetrics 


83.90 


5.56 


31 


General Medicine 


77.77 


9.26 


31 


Tropical Diseases 


80.16 


7.2ii 


31 


Practical 


77.16 


8.03 


31 
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TABLE B-8 

CORRELATIONS OF PRE-CURSO VARIABLJ^S WITH COURSE SCORES (CURSO 2) 29 <JJ < 31 











u 


G 
^1 






<1) 






■P 

' tH 




s 


S 


Verbal Aptitude 


057 


127 


Mathematical Aptitude 


Oi+7 


118 


English Reading 




221 


Anatoiqy 


1*51* 


367* 


Physiology 


186 


020 


Pharmacology 


105 


099 


Microbiology 


058 


205 


Biochemistry 


108 


032 


Pathology 


166 


195 


Surgery 


. 117 


232 


Obstetrics 


192 


191 


General Medicine 


201 


057 


Public Health 


137 


086 


Pediatrics 


135 


275 


Psychiatry 


108 


08I4 



*Corielation is significant at .05 level 
NOTE: Decimal is omitted 
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TABLE B -12 

CORRELATIONS OF PRE-CUR30 LICENSING VARIABLES WITH COURSE SCORE VARIABL: 

(CURSO 2) 13 ^ 1^ i. 31 
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5h ?h 
<U <L) 

Anatomy 266 3kh 

T'-yslol^^^ 635* 322 

Pharmacology 370 28^^ 

Pathology I56 OOU 

Surge ly 288 ' •29k 

Obstetrics 382* ^ 315 

General Medicine 08i+ 032 

Tropical Diseases 253 305 



^Correlation is significant at .05 level 
NOTE: Decimal is oinitted 
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TABLE B-15 

CORRELATIONS OF POST-CURSO VARIABLES WITH COURSE SCORE VAEIABLES 

(CURSO 2) 30 4 < 3:, 









n 














Anavoiny 
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Pharmaeciogy 
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Microti oloi?y 
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Pathology 
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Obstetrics 
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General Medicine 
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092 



*Correlation' is significai.t at .05 leve,l 
NOTE: Decimal is omitted 
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TAj3LE "b-l8 

INTERCORRELATION OF COURSE SCORE VARIABLES (CURSO 2) N = 26 



Id 



Midterm 2 ^ 723^ 



^Correlation is significant at .05 level 
NOTE: Decimal is omitted 
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TABLE B~19 

CORRELATIONS OF COURSE SCORE VARIABLIJS WITH POST-CURSO LICENSII^G VARIAj3LES 



(CURSO 2) 


27 < N ,< 30 
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088 


116 



^Correlation is significant at .05 level 
NOTE: Decimal is omitted 
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APPENDIX C 



UPR MEDICAL iH^OWLEDGE 
TEST SCORE PROFILES FOR SIX 
FOREIGN TRAINED PHYSICIANS 
WHO PARTICIPATED IN BOTH CURSOS 



• • PRE CURSO I 

POST CURSO I 
• • POST CURSO 2 
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Figure C-1: Test Profiles for Physician No, 393 
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• • PRE CURSO I 

30 -| • • POST CDRSO I 

• • POST CURSO 2 
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Figure 0-2: Test Profiles for Physician No. 296 
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Figure C-3: Test Profiles for Physician No. 356 
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• • PRE CURSO 1 

30n • POST CURSO I 

• • POST CURSO 2 
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Figure C-4: Test Profiles for Physician No. 360 
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^ ffl i ffl^fflfTiH i l l ^ Figure C-5: Test. Profiles for Physician No. 346 
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• • PRE CURSO i 

30 -| • • POST CURSO I 

• • P03TCURS02 



25- 



20- 



!5 



to 



% 



A 

i \i \ 





A 

/ \ 
/ \ 



/ 



\ 




>- 

:s 
o 

< 
z 
< 



o 
o 



CO 
X 



Q 

o 
o 
< 

< 



> 
O 
O 
«J 

o 
ffi 
o 
or 
o 



q: 



i 

UJ 



o 
o 

01 



o 
a 
o 
I 

2 



QC 
O 



CO 

o 

tx: 
h- 

UJ 

h- 

CO 
CD 
O 



111 ^ 

z o 

Ul UJ 

C9 Z 



GQ < 
UJ 

Q. X 



CO 


>- 


o 








rr 


< 






< 


X 




o 


a 


>• 


UJ 


CO 


CL. 


a 



ERIC 



Figure C-6: Test Profiles for Physician No, 303 



